A Template for Systematic Reviews and Meta-analyses
put your name here
put your institution hereyour.email.here@uni.edu
put coauthor’s name here
put coauthor’s institution herecoauthors.email.here@uni.edu
2021-02-08
This is a Rmd-template for protocols and reporting of systematic reviews and meta-analyses. It synthesizes three sources of standards:
The template is aimed at
- guiding the process of planning systematic review/meta-analyses
- providing a form for preregistration (enter your text, export as standalone HTML or PDF, upload as preregistration to any preferred repository)
We are aware that MARS targets aspects of reporting after the systematic review/meta-analysis is completed rather than decisions and reasoning in the planning phase as PRISMA-P and PROSPERO. MARS nevertheless provides a good framework to determine crucial points for systematic reviews/ meta-analyses to be addressed as early as in the planning phase.
Standards have been partially adapted. Click ‘show changes’ to see changes and reasons for the change.
| Standard | Implemented change | Reason |
|---|---|---|
| MARS | Left out the paper section “Abstract” | An abstract is important for reporting, however not, for planning and registering. |
| MARS | Left out the paper section “Results” and parts of “Discussion” | Specifications on how to report results is important for reporting, however not, for planning and registering. Prospective information on how results will be computed/ synthesized is preserved. |
| MARS | Left out “Protocol: List where the full protocol can be found” | This form serves as a preregistration form, a protocol will be generated later in the research process. |
| MARS | Left out “Give the place where the synthesis is registered and its registry number, if registered” | This template serves as a preregistration form. |
| PROSPERO | Left out non-mandatory fields or integrated them into the mandatory fields. | Avoiding too detailed specifications. All relevant information will be integrated. |
| PROSPERO | Left out some options in “Type and method of review” | Options left out are purely health/ medicine related. |
| PROSPERO | Left out “Health area of the review” | This field is purely health/ medicine related. |
| PRISMA-P | Left out “If registered, provide the name of the registry (such as PROSPERO) and registration number.” | This template serves as a preregistration form. |
1 General
1.1 Working Title
# avoiding markdown tables because they're not exactly the prettiest flower in the bunch
# set up the table
table_sources <- data.frame(Source = c("PRISMA-P", # first column will be always the same
"PROSPERO",
"MARS"),
Description = c(# PRISMA-P
"Identify the report as a protocol of a systematic review. If the protocol is for an update of a previous systematic review, identify as such.",
# PROSPERO
"Give the working title of the review, for example the one used for obtaining funding. Ideally the title should state succinctly the interventions or exposures being reviewed and the associated health or social problems. Where appropriate, the title should use the PI(E)COS structure to contain information on the Participants, Intervention (or Exposure) and Comparison groups, the Outcomes to be measured and Study designs to be included. For reviews in languages other than English, this field should be used to enter the title in the language of the review. This will be displayed together with the English language title.",
# MARS
"Title: State the research question and type of research synthesis (e.g., narrative synthesis, meta-analysis)."))
# produce table
knitr::kable(table_sources) %>%
kable_styling(fixed_thead = T, full_width = T) %>%
column_spec(1, bold = T) %>%
row_spec(0, background = "#ececec")| Source | Description |
|---|---|
| PRISMA-P | Identify the report as a protocol of a systematic review. If the protocol is for an update of a previous systematic review, identify as such. |
| PROSPERO | Give the working title of the review, for example the one used for obtaining funding. Ideally the title should state succinctly the interventions or exposures being reviewed and the associated health or social problems. Where appropriate, the title should use the PI(E)COS structure to contain information on the Participants, Intervention (or Exposure) and Comparison groups, the Outcomes to be measured and Study designs to be included. For reviews in languages other than English, this field should be used to enter the title in the language of the review. This will be displayed together with the English language title. |
| MARS | Title: State the research question and type of research synthesis (e.g., narrative synthesis, meta-analysis). |
Put your text here.
1.2 Type of Review
# avoiding markdown tables because they're not exactly the prettiest flower in the bunch
# set up the table
table_sources <- data.frame(Source = c("PRISMA-P", # first column will be always the same
"PROSPERO",
"MARS"),
Description = c(# PRISMA-P
"Not specified.",
# PROSPERO
"Type and method of review: Select the type of review and the review method from the lists below. Select the health area(s) of interest for your review.
* Meta-analysis
* Narrative synthesis
* Network meta-analysis
* Review of reviews
* Synthesis of qualitative studies
* Systematic review
* Other",
# MARS
"Not specified."))
# produce table
knitr::kable(table_sources) %>%
kable_styling(fixed_thead = T, full_width = T) %>%
column_spec(1, bold = T) %>%
row_spec(0, background = "#ececec")| Source | Description |
|---|---|
| PRISMA-P | Not specified. |
| PROSPERO |
Type and method of review: Select the type of review and the review method from the lists below. Select the health area(s) of interest for your review.
|
| MARS | Not specified. |
Put your text here.
1.3 Anticipated Start and Completion Date
# avoiding markdown tables because they're not exactly the prettiest flower in the bunch
# set up the table
table_sources <- data.frame(Source = c("PRISMA-P", # first column will be always the same
"PROSPERO",
"MARS"),
Description = c(# PRISMA-P
"Not specified.",
# PROSPERO
"Give the date when the systematic review commenced, or is expected to commence. Give the date by which the review is expected to be completed.",
# MARS
"Not specified."))
# produce table
knitr::kable(table_sources) %>%
kable_styling(fixed_thead = T, full_width = T) %>%
column_spec(1, bold = T) %>%
row_spec(0, background = "#ececec")| Source | Description |
|---|---|
| PRISMA-P | Not specified. |
| PROSPERO | Give the date when the systematic review commenced, or is expected to commence. Give the date by which the review is expected to be completed. |
| MARS | Not specified. |
Put your text here.
1.4 Stage of Synthesis
# avoiding markdown tables because they're not exactly the prettiest flower in the bunch
# set up the table
table_sources <- data.frame(Source = c("PRISMA-P", # first column will be always the same
"PROSPERO",
"MARS"),
Description = c(# PRISMA-P
"Not specified.",
# PROSPERO
"Indicate the stage of progress of the review by ticking the relevant Started and Completed boxes. Additional
information may be added in the free text box provided.
Please note: Reviews that have progressed beyond the point of completing data extraction at the time of
initial registration are not eligible for inclusion in PROSPERO. Should evidence of incorrect status and/or
completion date being supplied at the time of submission come to light, the content of the PROSPERO
record will be removed leaving only the title and named contact details and a statement that inaccuracies in
the stage of the review date had been identified.
This field should be updated when any amendments are made to a published record and on completion and
publication of the review. If this field was pre-populated from the initial screening questions then you are not
able to edit it until the record is published.
* The review has not yet started: [yes/no]
| Review stage | Started | Completed |
|:--------------------------------------------| :----:| :----:|
| Preliminary searches | Yes/No | Yes/No
| Piloting of the study selection process | Yes/No | Yes/No
| Formal screening of search results against eligibility criteria | Yes/No | Yes/No
| Data extraction | Yes/No | Yes/No
| Risk of bias (quality) assessment | Yes/No | Yes/No
| Data analysis | Yes/No | Yes/No
Provide any other relevant information about the stage of the review here (e.g. Funded proposal, protocol not
yet finalised).",
# MARS
"Not specified."))
# produce table
knitr::kable(table_sources, escape = F) %>%
kable_styling(fixed_thead = T, full_width = T) %>%
column_spec(1, bold = T) %>%
row_spec(0, background = "#ececec")| Source | Description | |||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| PRISMA-P | Not specified. | |||||||||||||||||||||
| PROSPERO |
Indicate the stage of progress of the review by ticking the relevant Started and Completed boxes. Additional information may be added in the free text box provided. Please note: Reviews that have progressed beyond the point of completing data extraction at the time of initial registration are not eligible for inclusion in PROSPERO. Should evidence of incorrect status and/or completion date being supplied at the time of submission come to light, the content of the PROSPERO record will be removed leaving only the title and named contact details and a statement that inaccuracies in the stage of the review date had been identified. This field should be updated when any amendments are made to a published record and on completion and publication of the review. If this field was pre-populated from the initial screening questions then you are not able to edit it until the record is published.
|
|||||||||||||||||||||
| MARS | Not specified. |
Put your text here.
1.5 Names, Affiliations, Contact
# avoiding markdown tables because they're not exactly the prettiest flower in the bunch
# set up the table
table_sources <- data.frame(Source = c("PRISMA-P", # first column will be always the same
"PROSPERO",
"MARS"),
Description = c(# PRISMA-P
"
* Provide name, institutional affiliation, e-mail address of all protocol authors; provide physical mailing address of corresponding author.
* Describe contributions of protocol authors and identify the guarantor of the review.",
# PROSPERO
"
* Named Contact: The named contact acts as the guarantor for the accuracy of the information presented in the register record.
* Named contact email: Give the electronic mail address of the named contact.
* Organisational affiliation of the review: Full title of the organisational affiliations for this review and website address if available. This field may be completed as 'None' if the review is not affiliated to any organisation.
* Review team members and their organisational affiliations: Give the personal details and the organisational affiliations of each member of the review team. Affiliation refers to groups or organisations to which review team members belong.",
# MARS
"Not specified."))
# produce table
knitr::kable(table_sources) %>%
kable_styling(fixed_thead = T, full_width = T) %>%
column_spec(1, bold = T) %>%
row_spec(0, background = "#ececec")| Source | Description |
|---|---|
| PRISMA-P |
|
| PROSPERO |
|
| MARS | Not specified. |
Put your text here.
1.6 Collaborators
# avoiding markdown tables because they're not exactly the prettiest flower in the bunch
# set up the table
table_sources <- data.frame(Source = c("PRISMA-P", # first column will be always the same
"PROSPERO",
"MARS"),
Description = c(# PRISMA-P
"Not specified.",
# PROSPERO
"Collaborators (name & affilitation) of individuals working on the review, but are not review team member.",
# MARS
"Not specified."))
# produce table
knitr::kable(table_sources) %>%
kable_styling(fixed_thead = T, full_width = T) %>%
column_spec(1, bold = T) %>%
row_spec(0, background = "#ececec")| Source | Description |
|---|---|
| PRISMA-P | Not specified. |
| PROSPERO | Collaborators (name & affilitation) of individuals working on the review, but are not review team member. |
| MARS | Not specified. |
Put your text here.
1.7 Amendments to previous versions
# avoiding markdown tables because they're not exactly the prettiest flower in the bunch
# set up the table
table_sources <- data.frame(Source = c("PRISMA-P", # first column will be always the same
"PROSPERO",
"MARS"),
Description = c(# PRISMA-P
"If the protocol represents an amendment of a previously completed or published protocol, identify as such and list changes; otherwise, state plan for documenting important protocol amendments.",
# PROSPERO
"Not specified.",
# MARS
"Not specified."))
# produce table
knitr::kable(table_sources) %>%
kable_styling(fixed_thead = T, full_width = T) %>%
column_spec(1, bold = T) %>%
row_spec(0, background = "#ececec")| Source | Description |
|---|---|
| PRISMA-P | If the protocol represents an amendment of a previously completed or published protocol, identify as such and list changes; otherwise, state plan for documenting important protocol amendments. |
| PROSPERO | Not specified. |
| MARS | Not specified. |
Put your text here.
1.8 Funding Sources, Sponsors and Their Roles
# avoiding markdown tables because they're not exactly the prettiest flower in the bunch
# set up the table
table_sources <- data.frame(Source = c("PRISMA-P", # first column will be always the same
"PROSPERO",
"MARS"),
Description = c(# PRISMA-P
"
* Indicate sources of financial or other support for the review.
* Provide name for the review funder and/or sponsor.
* Describe roles of funder(s), sponsor(s), and/or institution(s), if any, in developing the protocol.
",
# PROSPERO
"Funding sources/sponsors: Give details of the individuals, organizations, groups or other legal entities who take responsibility for initiating, managing, sponsoring and/or financing the review. Include any unique identification numbers assigned to the review by the individuals or bodies listed. If available, provide grant number(s).",
# MARS
"
* List all sources of monetary and in-kind funding support.
* State the role of funders in conducting the synthesis and deciding to publish the results, if any.
"))
# produce table
knitr::kable(table_sources) %>%
kable_styling(fixed_thead = T, full_width = T) %>%
column_spec(1, bold = T) %>%
row_spec(0, background = "#ececec")| Source | Description |
|---|---|
| PRISMA-P |
|
| PROSPERO | Funding sources/sponsors: Give details of the individuals, organizations, groups or other legal entities who take responsibility for initiating, managing, sponsoring and/or financing the review. Include any unique identification numbers assigned to the review by the individuals or bodies listed. If available, provide grant number(s). |
| MARS |
|
Put your text here.
1.9 Conflict of Interest
# avoiding markdown tables because they're not exactly the prettiest flower in the bunch
# set up the table
table_sources <- data.frame(Source = c("PRISMA-P", # first column will be always the same
"PROSPERO",
"MARS"),
Description = c(# PRISMA-P
"Not specified.",
# PROSPERO
"List any conditions that could lead to actual or perceived undue influence on judgements concerning the main topic investigated in the review.",
# MARS
"Describe possible conflicts of interest, including financial and other nonfinancial interests."))
# produce table
knitr::kable(table_sources) %>%
kable_styling(fixed_thead = T, full_width = T) %>%
column_spec(1, bold = T) %>%
row_spec(0, background = "#ececec")| Source | Description |
|---|---|
| PRISMA-P | Not specified. |
| PROSPERO | List any conditions that could lead to actual or perceived undue influence on judgements concerning the main topic investigated in the review. |
| MARS | Describe possible conflicts of interest, including financial and other nonfinancial interests. |
Put your text here.
2 Introduction
2.1 Rationale
# avoiding markdown tables because they're not exactly the prettiest flower in the bunch
# set up the table
table_sources <- data.frame(Source = c("PRISMA-P", # first column will be always the same
"PROSPERO",
"MARS"),
Description = c(# PRISMA-P
"Describe the rationale for the review in the context of what is already known.",
# PROSPERO
"Not specified.",
# MARS
"Problem: State the question or relation(s) under investigation, including
* Historical background, including previous syntheses and meta-analyses related to the topic
* Theoretical, policy, and/or practical issues related to the question or relation(s) of interest
* Populations and settings to which the question or relation(s) is relevant
* Rationale for
(a) choice of study designs,
(b) the selection and coding of outcomes,
(c) the selection and coding potential moderators or mediators of results
* Psychometric characteristics of outcome measures and other variables
"))
# produce table
knitr::kable(table_sources) %>%
kable_styling(fixed_thead = T, full_width = T) %>%
column_spec(1, bold = T) %>%
row_spec(0, background = "#ececec")| Source | Description |
|---|---|
| PRISMA-P | Describe the rationale for the review in the context of what is already known. |
| PROSPERO | Not specified. |
| MARS |
Problem: State the question or relation(s) under investigation, including
|
Put your text here.
2.2 Research Questions
# avoiding markdown tables because they're not exactly the prettiest flower in the bunch
# set up the table
table_sources <- data.frame(Source = c("PRISMA-P", # first column will be always the same
"PROSPERO",
"MARS"),
Description = c(# PRISMA-P
"Provide an explicit statement of the question(s) the review will address with reference to participants, interventions, comparators, and outcomes (PICO)",
# PROSPERO
"State the question(s) to be addressed by the review, clearly and precisely. Review questions may be specific or broad. It may be appropriate to break very broad questions down into a series of related more specific questions. Questions may be framed or refined using PI(E)COS where relevant.",
# MARS
"Objectives: State the hypotheses examined, indicating which were prespecified, including
* Question in terms of relevant participant characteristics (including animal populations), independent variables (experimental manipulations, treatments, or interventions), ruling out of possible confounding variables, dependent variables (outcomes, criterion), and other features of study designs
* Method(s) of synthesis and if meta-analysis was used, the specific methods used to integrate studies (e.g., effect-size metric, averaging method, the model used in homogeneity analysis)"))
# produce table
knitr::kable(table_sources) %>%
kable_styling(fixed_thead = T, full_width = T) %>%
column_spec(1, bold = T) %>%
row_spec(0, background = "#ececec")| Source | Description |
|---|---|
| PRISMA-P | Provide an explicit statement of the question(s) the review will address with reference to participants, interventions, comparators, and outcomes (PICO) |
| PROSPERO | State the question(s) to be addressed by the review, clearly and precisely. Review questions may be specific or broad. It may be appropriate to break very broad questions down into a series of related more specific questions. Questions may be framed or refined using PI(E)COS where relevant. |
| MARS |
Objectives: State the hypotheses examined, indicating which were prespecified, including
|
Put your text here.
3 Methods
3.1 Eligibility: Inclusion and Exclusion Criteria
# avoiding markdown tables because they're not exactly the prettiest flower in the bunch
# set up the table
table_sources <- data.frame(Source = c("PRISMA-P", # first column will be always the same
"PROSPERO",
"MARS",
"Added by authors"),
Description = c(# PRISMA-P
"Specify the study characteristics (such as PICO, study design, setting, time frame) and report characteristics (such as years considered, language, publication status) to be used as criteria for eligibility for the review.",
# PROSPERO
"Give details of the types of study (study designs) eligible for inclusion in the review. If there are no restrictions on the types of study design eligible for inclusion, or certain study types are excluded, this should be stated. The preferred format includes details of both inclusion and exclusion criteria.",
# MARS
"Describe the criteria for selecting studies, including
* Independent variables (e.g., experimental manipulations, types of treatments or interventions or predictor variables).
* Dependent variable (e.g., outcomes, in syntheses of clinical research including both potential benefits and potential adverse effects).
* Eligible study designs (e.g., methods of sampling or treatment assignment).
* Handling of multiple reports about the same study or sample, describing which are primary and handling of multiple measures using the same participants.
* Restrictions on study inclusion (e.g., by study age, language, location, or report type).
* Changes to the prespecified inclusion and exclusion criteria, and when these changes were made.
* Handling of reports that did not contain sufficient information to judge eligibility (e.g., lacking information about study design) and reports that did not include sufficient information for analysis (e.g., did not report numerical data about those outcomes).",
# added by authors
"Alternative approaches (to PICO) to describe study characteristics:
* SPIDER: relevant, when including qualitative research (https://doi.org/10.1177/1049732312452938)
* PICOS: Compared to PICO includes study design and reaches higher specifity (ISBN: 978-1-900640-47-3; https://www.york.ac.uk/media/crd/Systematic_Reviews.pdf)
* UTOS: Cronbach's classical framework (ISBN: 978-0875895253)"))
# produce table
knitr::kable(table_sources) %>%
kable_styling(fixed_thead = T, full_width = T) %>%
column_spec(1, bold = T) %>%
row_spec(0, background = "#ececec")| Source | Description |
|---|---|
| PRISMA-P | Specify the study characteristics (such as PICO, study design, setting, time frame) and report characteristics (such as years considered, language, publication status) to be used as criteria for eligibility for the review. |
| PROSPERO | Give details of the types of study (study designs) eligible for inclusion in the review. If there are no restrictions on the types of study design eligible for inclusion, or certain study types are excluded, this should be stated. The preferred format includes details of both inclusion and exclusion criteria. |
| MARS |
Describe the criteria for selecting studies, including
|
|
Added by authors |
Alternative approaches (to PICO) to describe study characteristics: SPIDER: relevant, when including qualitative research (https://doi.org/10.1177/1049732312452938) PICOS: Compared to PICO includes study design and reaches higher specifity (ISBN: 978-1-900640-47-3; https://www.york.ac.uk/media/crd/Systematic_Reviews.pdf) UTOS: Cronbach’s classical framework (ISBN: 978-0875895253) |
3.1.1 Inclusion criteria
PICOS
- Population:
- formal learning context (planned as a learning activity to improve writing skills)
- L1 (first language)
- years: 2003 and younger (rationale see ‘Search Strategy’)
- Intervention:
- system/ computer generated feedback
- formative feedback
- immediate feedback
- targeted at improving writing
- Comparison/ Control:
- none
- Outcome: improvement of text quality
- cohesion (organization)
- comprehension/ comprehensability
- lexical measures
- sentence structure (variety of sentences)
- overall development (awareness of purpose, task, and audience)
- (different!) overall quality scores
- specifity (Problems, solutions, locatization)
- Study Type
- empirical (quantitative and qualitative)
3.1.2 Exclusion criteria
PICOS
- Population:
- no formal learning context (e.g. novel writing; diary writing)
- not in the context of writing
- L2 learning (second language learning)
- Intervention:
- peer feedback
- summative feedback
- delayed feedback
- teacher feedback using computer tools (mediated feedback)
- targeted at improving content learning
- Comparison/ Control:
- not empirical
- Outcome:
- does not focus text quality
- conceptual content knowledge
- Study Type
- purely conceptual
3.2 Sources of Search: List and Rationale
# avoiding markdown tables because they're not exactly the prettiest flower in the bunch
# set up the table
table_sources <- data.frame(Source = c("PRISMA-P", # first column will be always the same
"PROSPERO",
"MARS"),
Description = c(# PRISMA-P
"Not specified.",
# PROSPERO
"Searches: State the sources that will be searched. Give the search dates, and any restrictions (e.g. language or publication period). Do NOT enter the full search strategy (it may be provided as a link or attachment).",
# MARS
"Describe all information sources:
* Databases searched (e.g., PsycINFO, ClinicalTrials.gov), including dates of coverage (i.e., earliest and latest records included in the search), and software and search platforms used
* Names of specific journals that were searched and the volumes checked
* Explanation of rationale for choosing reference lists if examined (e.g., other relevant articles, previous research
syntheses)
* Documents for which forward (citation) searches were conducted, stating why these documents were chosen
* Number of researchers contacted if study authors or individual researchers were contacted to find studies or to obtain more information about included studies, as well as criteria for making contact (e.g., previous relevant publications), and response rate
* Dates of contact if other direct contact searches were conducted such as contacting corporate sponsors or mailings to distribution lists
* Search strategies in addition to those above and the results of these searches
"))
# produce table
knitr::kable(table_sources) %>%
kable_styling(fixed_thead = T, full_width = T) %>%
column_spec(1, bold = T) %>%
row_spec(0, background = "#ececec")| Source | Description |
|---|---|
| PRISMA-P | Not specified. |
| PROSPERO | Searches: State the sources that will be searched. Give the search dates, and any restrictions (e.g. language or publication period). Do NOT enter the full search strategy (it may be provided as a link or attachment). |
| MARS |
Describe all information sources:
|
- specific target journals with search term
- all target journals are indexed in one of the data bases (Scopus or Web of Science)
- data bases:
- Scopus
- Web of Science
- first 100 results from google scholar
- backwards search from articles:
- After screening abstracts and titles: We use the three latest synthesis articles (reviews or meta-analyses) and screen their references
- preprint server
- PsyArXiv
- metaArXiv
3.3 Search Strategy
# avoiding markdown tables because they're not exactly the prettiest flower in the bunch
# set up the table
table_sources <- data.frame(Source = c("PRISMA-P", # first column will be always the same
"PROSPERO",
"MARS"),
Description = c(# PRISMA-P
"Present draft of search strategy to be used for at least one electronic database, including planned limits, such that it could be repeated.",
# PROSPERO
"URL to search strategy: Give a link to a published pdf/word document detailing either the search strategy or an example of a search strategy for a specific database if available (including the keywords that will be used in the search strategies), or upload your search strategy. Do NOT provide links to your search results. Alternatively, upload your search strategy to CRD in pdf format. Please note that by doing so you are consenting to the file being made publicly accessible.",
# MARS
"Describe all information sources: Search strategies of electronic searches, such that they could be repeated (e.g., include the search terms used, Boolean connectors, fields searched, explosion of terms)."))
# produce table
knitr::kable(table_sources) %>%
kable_styling(fixed_thead = T, full_width = T) %>%
column_spec(1, bold = T) %>%
row_spec(0, background = "#ececec")| Source | Description |
|---|---|
| PRISMA-P | Present draft of search strategy to be used for at least one electronic database, including planned limits, such that it could be repeated. |
| PROSPERO | URL to search strategy: Give a link to a published pdf/word document detailing either the search strategy or an example of a search strategy for a specific database if available (including the keywords that will be used in the search strategies), or upload your search strategy. Do NOT provide links to your search results. Alternatively, upload your search strategy to CRD in pdf format. Please note that by doing so you are consenting to the file being made publicly accessible. |
| MARS | Describe all information sources: Search strategies of electronic searches, such that they could be repeated (e.g., include the search terms used, Boolean connectors, fields searched, explosion of terms). |
Search String: PICO
- Participants:
NOT (peer OR medic* OR neural-network OR health* OR care*) - Intervention:
((computer* OR automat*) AND (writ* AND (argument* OR essay* OR summary OR exposit* OR expla*)) AND (feedback OR evaluat* OR assess* OR scor*) - Control: none
- Outcome: none
Timespan:
2003 - 2020
Around 2003 several programs were either established of got a major upgrade (see below). Before this timeframe, programs were rarely powered by machine learning (or related) algorythms. They had less or different range of functions and were thus different programs. We therefore choose this timeframe to make make analyses more comparable and reduce bias.
Examples:
- MyAccess: established in 2003
- Project Essay Grade: Bought by Measurement, Inc. in 2002/2003 and subsequently utilized AI scoring engine
- Citerion: major upgrade of scoring engine from e-rater V1.3 to e-rater V2.0 in 2004 (Attali & Burstein, 2004)
- Summary Street: established between 2002 and 2004 (Wade-Stein & Kintsch, 2004)
3.4 Data Management Tools Used
# avoiding markdown tables because they're not exactly the prettiest flower in the bunch
# set up the table
table_sources <- data.frame(Source = c("PRISMA-P", # first column will be always the same
"PROSPERO",
"MARS"),
Description = c(# PRISMA-P
"Describe the mechanism(s) that will be used to manage records and data throughout the review.",
# PROSPERO
"Not specified.",
# MARS
"Not specified."))
# produce table
knitr::kable(table_sources) %>%
kable_styling(fixed_thead = T, full_width = T) %>%
column_spec(1, bold = T) %>%
row_spec(0, background = "#ececec")| Source | Description |
|---|---|
| PRISMA-P | Describe the mechanism(s) that will be used to manage records and data throughout the review. |
| PROSPERO | Not specified. |
| MARS | Not specified. |
Put your text here.
3.5 Selection of Studies
# avoiding markdown tables because they're not exactly the prettiest flower in the bunch
# set up the table
table_sources <- data.frame(Source = c("PRISMA-P", # first column will be always the same
"PROSPERO",
"MARS"),
Description = c(# PRISMA-P
"State the process that will be used for selecting studies (such as two independent reviewers) through each phase of the review (that is, screening, eligibility and inclusion in meta-analysis).",
# PROSPERO
"Data extraction (selection and coding): Describe how studies will be selected for inclusion. State what data will be extracted or obtained. State how this will be done and recorded.",
# MARS
"Describe the process for deciding which studies would be included in the syntheses and/or included in the meta-analysis, including
* Document elements (e.g., title, abstract, full text) used to make decisions about inclusion or exclusion from the synthesis at each step of the screening process
* Qualifications (e.g., training, educational or professional status) of those who conducted each step in the study selection process, stating whether each step was conducted by a single person or in duplicate as well as an explanation of how reliability was assessed if one screener was used and how disagreements were resolved if multiple were used."))
# produce table
knitr::kable(table_sources) %>%
kable_styling(fixed_thead = T, full_width = T) %>%
column_spec(1, bold = T) %>%
row_spec(0, background = "#ececec")| Source | Description |
|---|---|
| PRISMA-P | State the process that will be used for selecting studies (such as two independent reviewers) through each phase of the review (that is, screening, eligibility and inclusion in meta-analysis). |
| PROSPERO | Data extraction (selection and coding): Describe how studies will be selected for inclusion. State what data will be extracted or obtained. State how this will be done and recorded. |
| MARS |
Describe the process for deciding which studies would be included in the syntheses and/or included in the meta-analysis, including
|
Put your text here.
3.6 Method of Extracting Data & Information (from Reports)
# avoiding markdown tables because they're not exactly the prettiest flower in the bunch
# set up the table
table_sources <- data.frame(Source = c("PRISMA-P", # first column will be always the same
"PROSPERO",
"MARS"),
Description = c(# PRISMA-P
"Describe planned method of extracting data from reports (such as piloting forms, done independently, in duplicate), any processes for obtaining and confirming data from investigators.",
# PROSPERO
"Not specified.",
# MARS
"Describe methods of extracting data from reports, including
* Variables for which data were sought and the variable categories.
* Qualifications of those who conducted each step in the data extraction process, stating whether each step was conducted by a single person or in duplicate and an explanation of how reliability was assessed if one screener was used and how disagreements were resolved if multiple screeners were used as well as whether data coding forms, instructions for completion, and the data (including metadata) are available, stating where they can be found (e.g., public registry, supplemental materials)."))
# produce table
knitr::kable(table_sources) %>%
kable_styling(fixed_thead = T, full_width = T) %>%
column_spec(1, bold = T) %>%
row_spec(0, background = "#ececec")| Source | Description |
|---|---|
| PRISMA-P | Describe planned method of extracting data from reports (such as piloting forms, done independently, in duplicate), any processes for obtaining and confirming data from investigators. |
| PROSPERO | Not specified. |
| MARS |
Describe methods of extracting data from reports, including
|
Put your text here.
3.7 List and Description of Data and Information Extracted
# avoiding markdown tables because they're not exactly the prettiest flower in the bunch
# set up the table
table_sources <- data.frame(Source = c("PRISMA-P", # first column will be always the same
"PROSPERO",
"MARS"),
Description = c(# PRISMA-P
"
* List and define all variables for which data will be sought (such as PICO items, funding sources), any pre-planned data assumptions and simplifications
* List and define all outcomes for which data will be sought, including prioritization of main and additional outcomes, with rationale",
# PROSPERO
"
* Condition or domain being studied: Give a short description of the disease, condition or healthcare domain being studied. This could include health and wellbeing outcomes.
* Participants/population: Give summary criteria for the participants or populations being studied by the review. The preferred format includes details of both inclusion and exclusion criteria.
* Intervention(s), exposure(s): Give full and clear descriptions or definitions of the nature of the interventions or the exposures to be reviewed.
* Comparator(s)/control: Where relevant, give details of the alternatives against which the main subject/topic of the review will be compared (e.g. another intervention or a non-exposed control group). The preferred format includes details of both inclusion and exclusion criteria.
* Main and additional outcome(s): Give the pre-specified main (most important) outcomes of the review, including details of how the outcome is defined and measured and when these measurement are made, if these are part of the review inclusion criteria.
* Measures of effect: Please specify the effect measure(s) for you main outcome(s) e.g. relative risks, odds ratios, risk difference,
and/or 'number needed to treat.",
# MARS
"Not specified."))
# produce table
knitr::kable(table_sources) %>%
kable_styling(fixed_thead = T, full_width = T) %>%
column_spec(1, bold = T) %>%
row_spec(0, background = "#ececec")| Source | Description |
|---|---|
| PRISMA-P |
|
| PROSPERO |
|
| MARS | Not specified. |
3.8 Effect Size Transformation from Individual Studies
# avoiding markdown tables because they're not exactly the prettiest flower in the bunch
# set up the table
table_sources <- data.frame(Source = c("PRISMA-P", # first column will be always the same
"PROSPERO",
"MARS"),
Description = c(# PRISMA-P
"Not specified.",
# PROSPERO
"Not specified.",
# MARS
"Describe the statistical methods for calculating effect sizes, including the metric(s) used (e.g., correlation coefficients, differences in means, risk ratios) and formula(s) used to calculate effect sizes."))
# produce table
knitr::kable(table_sources) %>%
kable_styling(fixed_thead = T, full_width = T) %>%
column_spec(1, bold = T) %>%
row_spec(0, background = "#ececec")| Source | Description |
|---|---|
| PRISMA-P | Not specified. |
| PROSPERO | Not specified. |
| MARS | Describe the statistical methods for calculating effect sizes, including the metric(s) used (e.g., correlation coefficients, differences in means, risk ratios) and formula(s) used to calculate effect sizes. |
We plan to extract means, standard deviatoins and sample sizes from studies and convert them into standardized mean differences.
library(metafor)
# fabricated example data to test functionality of code
ex_df <- data.frame(study = c("Author 1 (2020)", "Author 2 (2019)",
"Author 3 (2019)", "Author 4 (2018)"),
m_tc = rnorm(4, 4, 1), # mean treatment group
m_cc = rnorm(4, 3.5, 1), # mean control group
sd_tc = rnorm(4, 1, .1), # SD treatment group
sd_cc = rnorm(4, 1, .1), # SD control group
n_tc = c(22, 25, 44, 32), # n treatment group
n_cc = c(22, 25, 40, 33)) # n control group
# calculate effect sizes as SMD
ex_df <- escalc(measure="SMD", # standardized mean difference
n1i = n_tc, n2i = n_cc, # group sizes
m1i = m_tc, m2i = m_cc, # means
sd1i = sd_tc, sd2i = sd_cc, # standard deviations
data=ex_df)
ex_df## study m_tc m_cc sd_tc sd_cc n_tc n_cc yi
## 1 Author 1 (2020) 3.340708 3.690844 0.9045304 0.9391957 22 22 -0.3729
## 2 Author 2 (2019) 3.669681 2.762059 1.1190937 1.1508069 25 25 0.7871
## 3 Author 3 (2019) 3.452128 3.726849 0.9162855 0.9214779 44 40 -0.2963
## 4 Author 4 (2018) 4.353792 2.529652 0.9243784 1.0362180 32 33 1.8339
## vi
## 1 0.0925
## 2 0.0862
## 3 0.0482
## 4 0.08743.9 Risk of Bias in Individual Studies
# avoiding markdown tables because they're not exactly the prettiest flower in the bunch
# set up the table
table_sources <- data.frame(Source = c("PRISMA-P", # first column will be always the same
"PROSPERO",
"MARS",
"Added by authors"),
Description = c(# PRISMA-P
"Describe anticipated methods for assessing risk of bias of individual studies, including whether this will be done at the outcome or study level, or both; state how this information will be used in data synthesis.",
# PROSPERO
"Risk of bias (quality) assessment: Describe the method of assessing risk of bias or quality assessment. State which characteristics of the studies will be assessed and any formal risk of bias tools that will be used.",
# MARS
"Describe any methods used to assess risk to internal validity in individual study results, including
* Risks assessed and criteria for concluding risk exists or does not exist.
* Methods for including risk to internal validity in the decisions to synthesize of the data and the interpretation of results.",
# added by authors
"Describe how the quality of original studies are rated. E.g. by 'The Study Design and Implementation Assessment Device (Study DIAD)': https://doi.org/10.1037/1082-989X.13.2.130 "))
# produce table
knitr::kable(table_sources) %>%
kable_styling(fixed_thead = T, full_width = T) %>%
column_spec(1, bold = T) %>%
row_spec(0, background = "#ececec")| Source | Description |
|---|---|
| PRISMA-P | Describe anticipated methods for assessing risk of bias of individual studies, including whether this will be done at the outcome or study level, or both; state how this information will be used in data synthesis. |
| PROSPERO | Risk of bias (quality) assessment: Describe the method of assessing risk of bias or quality assessment. State which characteristics of the studies will be assessed and any formal risk of bias tools that will be used. |
| MARS |
Describe any methods used to assess risk to internal validity in individual study results, including
|
| Added by authors | Describe how the quality of original studies are rated. E.g. by ‘The Study Design and Implementation Assessment Device (Study DIAD)’: https://doi.org/10.1037/1082-989X.13.2.130 |
Put your text here.
4 Results
4.1 Strategy for Data Synthesis
# avoiding markdown tables because they're not exactly the prettiest flower in the bunch
# set up the table
table_sources <- data.frame(Source = c("PRISMA-P", # first column will be always the same
"PROSPERO",
"MARS"),
Description = c(# PRISMA-P
"
* Describe criteria under which study data will be quantitatively synthesised.
* If data are appropriate for quantitative synthesis, describe planned summary measures, methods of handling data and methods of combining data from studies, including any planned exploration of consistency (such as I2, Kendall’s τ).
* If quantitative synthesis is not appropriate, describe the type of summary planned.",
# PROSPERO
"Strategy for data synthesis: Provide details of the planned synthesis including a rationale for the methods selected. This must not be generic text but should be specific to your review and describe how the proposed analysis will be applied to your data.",
# MARS
"Describe narrative and statistical methods used to compare studies. If meta-analysis was conducted, describe the methods used to combine effects across studies and the model used to estimate the heterogeneity of the effects sizes (e.g., a fixed-effect, random-effects model robust variance estimation), including
* Rationale for the method of synthesis.
* Methods for weighting study results.
* Methods to estimate imprecision (e.g., confidence or credibility intervals) both within and between studies.
* Description of all transformations or corrections (e.g., to account for small samples or unequal group numbers) and adjustments (e.g., for clustering, missing data, measurement artifacts, or construct-level relationships) made to the data and justification for these.
* Additional analyses (e.g., subgroup analyses, meta-regression), including whether each analysis was prespecified or post hoc.
* Selection of prior distributions and assessment of model fit if Bayesian analyses were conducted.
* Name and version number of computer programs used for the analysis.
* Statistical code and where it can be found (e.g., a supplement)."))
# produce table
knitr::kable(table_sources) %>%
kable_styling(fixed_thead = T, full_width = T) %>%
column_spec(1, bold = T) %>%
row_spec(0, background = "#ececec")| Source | Description |
|---|---|
| PRISMA-P |
|
|
PROSPERO |
Strategy for data synthesis: Provide details of the planned synthesis including a rationale for the methods selected. This must not be generic text but should be specific to your review and describe how the proposed analysis will be applied to your data. |
|
MARS |
Describe narrative and statistical methods used to compare studies. If meta-analysis was conducted, describe the methods used to combine effects across studies and the model used to estimate the heterogeneity of the effects sizes (e.g., a fixed-effect, random-effects model robust variance estimation), including Rationale for the method of synthesis. Methods for weighting study results. Methods to estimate imprecision (e.g., confidence or credibility intervals) both within and between studies. Description of all transformations or corrections (e.g., to account for small samples or unequal group numbers) and adjustments (e.g., for clustering, missing data, measurement artifacts, or construct-level relationships) made to the data and justification for these. Additional analyses (e.g., subgroup analyses, meta-regression), including whether each analysis was prespecified or post hoc. Selection of prior distributions and assessment of model fit if Bayesian analyses were conducted. Name and version number of computer programs used for the analysis. Statistical code and where it can be found (e.g., a supplement). |
library(dmetar)
library(tibble)
library(plotly)
# create empty data.frame to be filled
results.power <- data.frame(k = numeric(),
d = numeric(),
power = numeric())
for (k in 5:40) { # loop from k=5 to 40 studies
for (d in c(.25, .30, .35)) {
# compute power analyses for each k (random effects model)
tmp <- power.analysis(d=d, # expected effect size
k=k, # expected number of studies
n1=25, # expected mean group size in treatment group
n2=25, # expected mean group size in control group
p=0.05, # alpha level
heterogeneity = "moderate") # expected heterogeneity
results.power <- add_row(results.power,
k = k,
d = d,
power = tmp$Power)
}
}
# plot results
plot_ly(data = results.power,
x = ~k,
y = ~power,
hovertemplate = "<b>k:</b> %{x} <br /><b>power:</b> %{y}",
type="scatter",
color=~as.factor(d),
mode="lines+markers",
split=~as.factor(d)) %>%
add_lines(x = c(5, 40),
y = c(.8, .8),
inherit = F,
name = "80% power")The results indicate at least 80% power for 9, 12 or 17 included studies under different effect size assumptions. We take on a conservative approach assuming Cohen’s \(d=.25\) and thus will test the synthesized effect size when we are able to include 17 studies.
## k d power
## 1 5 0.25 0.3314842
## 2 5 0.30 0.4464151
## 3 5 0.35 0.5655469
## 4 6 0.25 0.3856321
## 5 6 0.30 0.5157014
## 6 6 0.35 0.6435011
## 7 7 0.25 0.4375575
## 8 7 0.30 0.5790673
## 9 7 0.35 0.7103232
## 10 8 0.25 0.4869204
## 11 8 0.30 0.6362987
## 12 8 0.35 0.7666693
## 13 9 0.25 0.5334934
## 14 9 0.30 0.6874399
## 15 9 0.35 0.8135264
## 16 10 0.25 0.5771421
## 17 10 0.30 0.7327163
## 18 10 0.35 0.8520299
## 19 11 0.25 0.6178080
## 20 11 0.30 0.7724747
## 21 11 0.35 0.8833411
## 22 12 0.25 0.6554939
## 23 12 0.30 0.8071352
## 24 12 0.35 0.9085706
## 25 13 0.25 0.6902505
## 26 13 0.30 0.8371559
## 27 13 0.35 0.9287337
## 28 14 0.25 0.7221656
## 29 14 0.30 0.8630056
## 30 14 0.35 0.9447297
## 31 15 0.25 0.7513544
## 32 15 0.30 0.8851455
## 33 15 0.35 0.9573354
## 34 16 0.25 0.7779518
## 35 16 0.30 0.9040157
## 36 16 0.35 0.9672092
## 37 17 0.25 0.8021056
## 38 17 0.30 0.9200272
## 39 17 0.35 0.9749003
## 40 18 0.25 0.8239713
## 41 18 0.30 0.9335569
## 42 18 0.35 0.9808606
## 43 19 0.25 0.8437078
## 44 19 0.30 0.9449457
## 45 19 0.35 0.9854577
## 46 20 0.25 0.8614735
## 47 20 0.30 0.9544982
## 48 20 0.35 0.9889878
## 49 21 0.25 0.8774243
## 50 21 0.30 0.9624839
## 51 21 0.35 0.9916876
## 52 22 0.25 0.8917112
## 53 22 0.30 0.9691388
## 54 22 0.35 0.9937443
## 55 23 0.25 0.9044786
## 56 23 0.30 0.9746685
## 57 23 0.35 0.9953056
## 58 24 0.25 0.9158639
## 59 24 0.30 0.9792506
## 60 24 0.35 0.9964867
## 61 25 0.25 0.9259959
## 62 25 0.30 0.9830375
## 63 25 0.35 0.9973774
## 64 26 0.25 0.9349955
## 65 26 0.30 0.9861596
## 66 26 0.35 0.9980472
## 67 27 0.25 0.9429745
## 68 27 0.30 0.9887275
## 69 27 0.35 0.9985493
## 70 28 0.25 0.9500365
## 71 28 0.30 0.9908350
## 72 28 0.35 0.9989247
## 73 29 0.25 0.9562765
## 74 29 0.30 0.9925609
## 75 29 0.35 0.9992047
## 76 30 0.25 0.9617814
## 77 30 0.30 0.9939714
## 78 30 0.35 0.9994129
## 79 31 0.25 0.9666306
## 80 31 0.30 0.9951221
## 81 31 0.35 0.9995675
## 82 32 0.25 0.9708959
## 83 32 0.30 0.9960589
## 84 32 0.35 0.9996820
## 85 33 0.25 0.9746425
## 86 33 0.30 0.9968204
## 87 33 0.35 0.9997666
## 88 34 0.25 0.9779291
## 89 34 0.30 0.9974383
## 90 34 0.35 0.9998290
## 91 35 0.25 0.9808085
## 92 35 0.30 0.9979389
## 93 35 0.35 0.9998749
## 94 36 0.25 0.9833280
## 95 36 0.30 0.9983438
## 96 36 0.35 0.9999087
## 97 37 0.25 0.9855300
## 98 37 0.30 0.9986708
## 99 37 0.35 0.9999334
## 100 38 0.25 0.9874524
## 101 38 0.30 0.9989345
## 102 38 0.35 0.9999515
## 103 39 0.25 0.9891288
## 104 39 0.30 0.9991469
## 105 39 0.35 0.9999648
## 106 40 0.25 0.9905891
## 107 40 0.30 0.9993177
## 108 40 0.35 0.9999744In case we are able to include enough studies we will calculate a random effects model. This is an example of the code with fabricated example data.
# calculating random effects model
rem <- rma(yi, # effect sizes
vi, # variances
data=ex_df)
summary(rem)##
## Random-Effects Model (k = 4; tau^2 estimator: REML)
##
## logLik deviance AIC BIC AICc
## -4.3768 8.7535 12.7535 10.9508 24.7535
##
## tau^2 (estimated amount of total heterogeneity): 1.0024 (SE = 0.8824)
## tau (square root of estimated tau^2 value): 1.0012
## I^2 (total heterogeneity / total variability): 92.99%
## H^2 (total variability / sampling variability): 14.27
##
## Test for Heterogeneity:
## Q(df = 3) = 41.6975, p-val < .0001
##
## Model Results:
##
## estimate se zval pval ci.lb ci.ub
## 0.4818 0.5198 0.9269 0.3540 -0.5370 1.5005
##
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
4.2 Moderators/ Subgroups
# avoiding markdown tables because they're not exactly the prettiest flower in the bunch
# set up the table
table_sources <- data.frame(Source = c("PRISMA-P", # first column will be always the same
"PROSPERO",
"MARS"),
Description = c(# PRISMA-P
"Describe any proposed additional analyses (such as sensitivity or subgroup analyses, meta-regression).",
# PROSPERO
"Analysis of subgroups or subsets: State any planned investigation of ‘subgroups’. Be clear and specific about which type of study or participant will be included in each group or covariate investigated. State the planned analytic approach.",
# MARS
"Not specified."))
# produce table
knitr::kable(table_sources) %>%
kable_styling(fixed_thead = T, full_width = T) %>%
column_spec(1, bold = T) %>%
row_spec(0, background = "#ececec")| Source | Description |
|---|---|
| PRISMA-P | Describe any proposed additional analyses (such as sensitivity or subgroup analyses, meta-regression). |
| PROSPERO | Analysis of subgroups or subsets: State any planned investigation of ‘subgroups’. Be clear and specific about which type of study or participant will be included in each group or covariate investigated. State the planned analytic approach. |
| MARS | Not specified. |
Put your text here.
4.3 Assessment of Publication Bias
# avoiding markdown tables because they're not exactly the prettiest flower in the bunch
# set up the table
table_sources <- data.frame(Source = c("PRISMA-P", # first column will be always the same
"PROSPERO",
"MARS"),
Description = c(# PRISMA-P
"Specify any planned assessment of meta-bias(es) (such as publication bias across studies, selective reporting within studies)",
# PROSPERO
"Not specified.",
# MARS
"Describe risk of bias across studies, including
* Statement about whether
(a) unpublished studies and unreported data, or
(b) only published data were included in the synthesis and the rationale if only published data were used
* Assessments of the impact of publication bias (e.g., modeling of data censoring, trim-and-fill analysis)
* Results of any statistical analyses looking for selective reporting of results within studies"))
# produce table
knitr::kable(table_sources) %>%
kable_styling(fixed_thead = T, full_width = T) %>%
column_spec(1, bold = T) %>%
row_spec(0, background = "#ececec")| Source | Description |
|---|---|
| PRISMA-P | Specify any planned assessment of meta-bias(es) (such as publication bias across studies, selective reporting within studies) |
| PROSPERO | Not specified. |
| MARS |
Describe risk of bias across studies, including
|
Put your text here.
5 Discussion
5.1 Strength of Cumulative Evidence
# avoiding markdown tables because they're not exactly the prettiest flower in the bunch
# set up the table
table_sources <- data.frame(Source = c("PRISMA-P", # first column will be always the same
"PROSPERO",
"MARS"),
Description = c(# PRISMA-P
"Describe how the strength of the body of evidence will be assessed (such as [GRADE](https://www.gradeworkinggroup.org/)).",
# PROSPERO
"Not specified.",
# MARS
"Describe the generalizability (external validity) of conclusions, including • Implications for related populations, intervention variations, dependent (outcome) variables."))
# produce table
knitr::kable(table_sources) %>%
kable_styling(fixed_thead = T, full_width = T) %>%
column_spec(1, bold = T) %>%
row_spec(0, background = "#ececec")| Source | Description |
|---|---|
| PRISMA-P | Describe how the strength of the body of evidence will be assessed (such as GRADE). |
| PROSPERO | Not specified. |
| MARS | Describe the generalizability (external validity) of conclusions, including • Implications for related populations, intervention variations, dependent (outcome) variables. |
Put your text here.